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Agenda 


• Intro 

• Unified Telemetry Rationale and Desig 

• RainbowSix|Siege Telemetry 

• Postmortem 
. Q&A 



Rainbow Six|Siege 

• Competitive Multiplayer Shooter 

• Next Gen Only 

• Fully Destructible Environments 

• Check out our RealBlast Talk on GDC16 Vault 

• Heavily Multithreaded 
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Rainbow Six|Siege - Multiplayer 







Rainbow Six|Siege - Game Server 

• Dedicated Servers 

• Hosted on Microsoft's Xbox Live Cloud 

• Azure (mostly) 

• No Remote Debugging © 

• Outside Ubisoft Internal Network 


Problem 


• How to profile/debug a game that: 

• Runs client-side code on 10 machines 

• Runs server-side code on a cloud-hosted VM 

• Is a live online service 

• Developed and tested by hundreds of devs 

• At several locations worldwide 
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Old Workflow 

• Instrument/Record: 

• Logs 

• Profile Points 

• Functional Scopes 

• Function Callstacks 

• Memory Snapshots 

• Memory Alloc/Free 

• Network Replication 


• Network Bandwidth 

• Size of Buffers 

• Particles 

• Gameplay Events 

• Structured Data 

• Resource Activity 

• and more... 
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Old Workflow 

• Run the game locally 

• Play it yourself 

• Try to reproduce the issue 

• Investigate 

• Rinse & Repeat 


Old Workflow for multiplayer 

• Run multiple instances locally 

• "Play" some of them 

• Try to reproduce the issue 

• Investigate 

• Doesn't Work for profiling and timing 
related bugs © 


Telemetry Powered Workflow 


• Instrument/Record: 

• Logs 

• Profile Points 

• Functional Scopes 

• Function Callstacks 

• Memory Snapshots 

• Memory Alloc/Free 

• Network Replication 


Network Bandwidth 
Size of Buffers 
Particles 

Gameplay Events 
Structured Data 
Resource Activity 
and more... 


ink 
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Telemetry Powered Workflow 

• Always-on Recording 

• Every engine/tool instance 

• Every developer 

• Everywhere 

• Every Platform 

• Automatically Collected Remotely 

• Post Processed & Stored Centrally 


ink 
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6 HUBBLE AN EYE ON PRODUCTION 


A Web Browser | Q Calenda r [ FSO * Jre | » I 


Asset 6ank • Mongo Stats [ 0 Bu8d| 


S Refresh Jira 5 Refresh Mongo ? - 
160706 issues retrieved on 20/01/2016 10:47:10. 


More ▼ Clear® ★Favorites. 


* Stats fig Trends 


| ^Settings... ||p^B«portZ 


User 
51 mmandici 


EE- arcturus1801 
Gp- bcaionescu 
EE-- arcturus1801 
(j) aebratu 


HD- lon-Ovidiu An 
5) arcturus1801 


Platform 

PS4 

PS4 

PS4 

PS4 

PS4 

PS4 

PS4 

PS4 

PS4 

PS4 

PS4 

PS4 

PS4 

PS4 

PS4 

PS4 


Target 

Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 
Engine - Profile 


Changelist 

1649602 

1649602 

1649602 

1649602 

1649602 

1649602 

1649602 

1649602 

1649602 

1649602 

1649602 

1649602 

1649602 

1649602 

1649602 

1649602 


HOUSE 

HOUSE 

HOUSE 

HOUSE 

HOUSE 

HOUSE 

HOUSE 

OREGON 

OREGON 

OREGON 

OREGON 

OREGON 

HEREFORD BASE 

HEREFORD BASE 

HEREFORD BASE 

HEREFORD BASE 


PvP 


PvP 

PvP 

PvP 

PvP 

PvP 

PvP 

PvP 

PvP 

PvP 

PvP 

PvP 

PvP 

PvP 

PvP 

PvP 


Game Mode 
TDM - BOMB 
TDM - BOMB 
TDM - BOMB 
TDM - BOMB 
TDM - BOMB 
TDM - BOMB 
TDM - BOMB 
TDM - HOSTAGE 
TDM - HOSTAGE 
TDM - HOSTAGE 
TDM - HOSTAGE 
TDM - HOSTAGE 
TDM - SECURE AREA 
TDM - SECURE AREA 
TDM -SECURE AREA 
TDM - SECURE AREA 


Session Name 


Client 

Client 

Client 

Client 

Client 

Client 

Client 

Client 

Client 

Client 

Client 

Client 

Client 

Client 

Client 

Client 


False 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 


Play Time 

0:00:15.42 

0:00:15.405 

0:04:05.209 

0:04:05.256 

0:00:15.402 

0:00:15.414 

0:03:52.887 

0:03:06 034 

0:02:23.532 

0:03:49.29 

0:03:49.645 

0:03:49.645 

0:03:17.24 

0:03:24.929 

0:02:44.746 

0:02:25.825 


Date 

01/18/2016 03:34 
01/18/2016 03:36 
01/18/2016 03:46 
01/18/2016 03:47 
01/18/2016 03:51 
01/18/2016 03:54 
01/18/2016 04:03 
01/18/2016 04:19 
01/18/2016 04:20 
01/18/2016 04:21 
01/18/2016 04:35 
01/18/2016 04:38 
01/18/2016 04:47 
01/18/2016 04:49 
01/18/2016 04:51 
01/18/2016 05:06 


Median FPS 
75.06 


Below 60 FPS Files 


Id 


Continuous 


73.82 

74.11 

74.34 

74.49 

71.52 

73.68 

78.24 

76.45 


0.26 

0.36 

3.60 

2.86 


0.19 


79.23 


78.3! 


73.40 

78.42 

79.21 

76.72 


0.46 

5.19 

0.55 

0.16 

0.26 

0.80 

0.68 

0.61 

0.33 

0.26 

0.26 


5ef446e1-bdbc-1 ... 
9f3c7T17-bdbc-1... 
56dbe0a1-bdbe- ... 
6e71fbb6-bdbe-1 .. 
a1abb28d-bdbf-1 ... 
f 3932a25-bdbf-1 . . . 


e50Qbc01 -bdcO-. . . 
e489e647-bdc3-. . . 
5ae5a828-bdc3-. . . 
237b49fd-bdc3-1 ... 
c5725554-bdc5-. . . 
6083a 10b-bdc5-.. 
8f4245cb-bdc5-1 ... 
f313ab5c-bdc7-1... 
1 2f 3ca03-bdc8-1 . . . 


-•I FPS [T-vH P erformance | 


Histogram 
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6 HUBBLE AN EYE ON PRODUCTION 


Web Browser 

Calendar pq FSO y Jira 

% Bloomberg 

mm Asset Bank 


0 Mongo Stats 0 Build 


Cell: 

All 


Date: ▼ 
01/18/2016 


# Stats 


Trends 


Map: ▼ 
All 


Platform: ▼ 
PS4 


Target: ▼ 
Engine - Profile 


User 

Platform 

Target 

Changelist 

Map 

Game Type 

Game Mode 

©■■ mmandici 

PS4 

Engine - Profile 

1649602 

HOUSE 

PvP 

TDM - BOMB 

©■■ pmpaltenea 

PS4 

Engine - Profile 

1 649602 

HOUSE 

PvP 

TDM - BOMB 

©■■ aharalambie 

PS4 

Engine - Profile 

1649602 

HOUSE 

PvP 

TDM - BOMB 

©■■ mmandici 

PS4 

Engine - Profile 

1649602 

HOUSE 

PvP 

TDM - BOMB 

©■■ ancturusISOI 

PS4 

Engine - Profile 

1649602 

HOUSE 

PvP 

TDM - BOMB 

©■• bcaionescu 

PS4 

Engine - Profile 

1649602 

HOUSE 

PvP 

TDM - BOMB 

©■■ ancturusISOI 

PS4 

Engine - Profile 

1649602 

HOUSE 

PvP 

TDM - BOMB 

©■• aebratu 

PS4 

Engine - Profile 

1649602 

OREGON 

PvP 

TDM - HOSTAGE 





r Jndex 

tags 

# threaded 

r _kl 
r _type 

f message_string 
r session_id 
O time 
r type 


i 


r6telemetrylogs 



Selected Fields 
9 line 


1000 000 

r file 

i 

800.000 

r message 

600 000 

r category 

o 

400.000 
200 000 

r username 


0 

Available Fields 
Popular 

□ 



August 12th 2015. 15 43 45 831 - October 1 1th 2015. 15 43 45 831 — bv dav 


2015-09-12 

time per day 


Time ^ 


message 


October 9th 2015, 15:14:11.448 [DisabledComponentUpdated] [QueueComponent] Requested update for 
Component Acti veGroundDrone which was inactive. Owner -{object : 
UNKNOWN id-OxfOOOa027 class-€ntity} 


» October 9th 2015, 12:40:19.171 


Allocation 

Type [0] peak allocation [9] waste [247] 

Type [1] peak allocation [0] waste [8] 

Type [2] peak allocation [0] waste [0] 

Type [3] peak allocation [116] waste [4384] 

- at illocati Btc fWil 


e:\rambow6\osborn\raain 
\framework source sci 
mi tar \engi ne\ani mat i o 
n\atom\atominstancepoo1 . cpp 


October 9th 2015, 12:40:19.168 This class has been tagged with the skipreleaseresources mold 
attribute. ReleaseResourcesO should not be called. 

1. Check if the class should really have this attribute(see 
progrmnmer guide for guidelines). 

2. If still needed, add the attribute to the referenced property. 


e : \r ai nbow6\os bor n \mai n 
\framework\source\sci 
mitar\engine\debug\re 
pi ay\debugr epl aydebug 
displaytrack. cpp 


► October 9th 2015, 12:40:18.055 Shut down () 


e : \r ai nbow6\os bor n \mai n 
\framework\source\sci 
mi tar \gr aphi c\ei kon\e 
kshaderdatabasemanager . cpp 


» October 9th 2015, 12:40:18.051 Pilot shutdown 


e : \r ai nbow6\os bor n \mai n 
\framework\source\sci 
mi t ar \engi ne\ai \pi 1 ot 


12,914,781 hits 


.li._iilli_ nil. ■■■_. ■■!■■■ ■ 1.1 lill. ■! 


username 

mdepascale 


sorin. ivascu 


731 sorin.ivascu 


412 sorin.ivascu 


476 sorin.ivascu 
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<> 


kibana 







r6 telemetry log 


Selected Fields 


line 

tile 

message 

category 


1 ooo. 
800 
600 , 
400 , 
200 , 


username 


i § i § i 
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jtGEAR STUDIO No semon active File Edit View Window Help Plugins 






▼ 19847bfd - bb5f - 1 Ie5-ac4§-00d 9dld0f0b5_0 estambulitchi PS4 Engine - Profile C1647491_D260330_SO_84 55798 BUC_PERF_CAPTURES round 0 • 

▼ Marker Telemetry] [RealB last: Input ]:DaTiageType_^leeempactal(3c=19. HM[Reai 

▼ Scoped Event 

Game State] Ga meState P by ing stacked 


[Game State] GameState Playing activated 

[Te le metry ] st ream ing / appO fl 0 16 .61.15 -0 3 .16 .10 _sbw_fra 


▼ CPU Frames 50. .00 Ms (Explicit zoom } 



6.00 

▼ C PU U pdates 50 0 0 Ms ( E* P*** zoom } 
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3 EAR STUDIO 


2015.10.09-15^346_stow_fra.„ 


View Window Help Plugins 


Frame Threshold (ms) 


~ 92,24 3 Events processed 29.7 Fps 
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£ Memento 

File E<® View Tools Window Debug 
Explot er 

PVP01_Houtexbox_V2 nemttalt 

B U FVP01_Housexbox_V2 memstats 
16/03/2015 1026:26 AM 
B ggj Cel Snapshots (2) Unknown 
j— ♦ Snapshot 01 

+ GraphtcResources Snapshot 01 
I «£) 14/04/2015 2:45:46 PM 
B §£$ Cel Snapshots (4) Unknown 

♦ SnapShot 01 

♦ GraphtcResources SnapShot 01 

♦ SnapShot 02 

♦ GraphtcResources SnapShot 02 
B 14/04/2015 4:03:51 PM 

B H Cel Snapshots (4) Unknown 

♦ SnapShot 01 

♦ GraphtcResources SnapShot 01 

♦ Snapshot 02 

^ GraphtcResources SnapShot 02 
23/01/2015 1 1.21 .10 AM 


5.573.195.594 

2.076.066.080 


1.246.072.191 

0 

1.289.992.072 


4.790.138.535 

1.266.069.921 

4.891.367.946 

1.276.672.112 


9 Cel Snapshots (8) Unknown > 

♦ SnapShot 01 5.203.799.012 

♦ GraphtcResources SnapShot 01 1.454.153.465 

♦ SnapShot 02 5.288.395.068 

♦ GraphtcResources SnapShot 02 1.475.085.111 

♦ SnapShot 03 5.324.588.348 

♦ GraphtcResources SnapShot 03 1 .481 .41 2.263 

♦ SnapShot 04 5.369.505.786 

♦ GraphtcResources SnapShot 04 1.484.838.729 

Snapshots taken in 
one play session 


4 SnapShot 04 

[> 

Name 


SubT ree * 


napShot 04 

Sound 

Data 

5.369.505,786 1 
2.776.940,074 
852.009.860 
848.324.436 

— 

100 0 X 
52.1 X 
16.0 X 
15.7* 

41 ,497,484 

2,940,023 

382,330 

13,256,626 

2190674 

607050 

2904 

924459 



H Entity 

494.417.620 


58 4* 

6.936.019 

554802 



H Animation 

135.071.268 

■ 

16 0* 

1.711.889 

23365 



-l MeshShape 

65.987.404 

■ 

7.8* 

1 .008.731 

8152 



E 

3 Physic 

53.853,648 


81 5* 

869,744 

53 



E 

MeshShape 

12,064,156 

m 

18.4* 

135.448 

6964 




m_Vertices(uibiVect. . . 

7,306,1 1 2 

wmam 

60.5* 

90,768 

1135 




m_l ndicesT nanglesf 

3.335.468 


27.8* 

18,530 

1135 




< MeshShape > 

799,280 

i 

6.5* 

18.400 

1135 




m_l ndicesMatenal(u 

591.040 

i 

4 9* 

7,734 

1135 




- MeshShapeWrapper 

18.160 


02* 

0 

1135 




m_M aterials(R etere 

12.608 


01 * 

16 

1135 

Snapshot 



n\_0 verrideColliswn 

1.488 


00* 

0 

154 

tags 


i 

3- SimpleString Template 

69.600 


01 * 

3,539 

1135 



H D estructionD ata 

34.398,912 

i 

4 1 * 

515.034 

31715 



l Mesh 

25.439,496 

i 

29 X 

1,034,822 

5595 



l World 

25,014,784 

i 

3.0* 

279,560 

1701 



S) Procedural^ esh 
±l Init 

+1 AtomGraph 
B LocafeationComponent 
i+l S plash FX 

+l- MergedDestructionHierarchy 
:+] Material 
(B TextureMap 
CB LocakzationPeckage 
3 Skeleton 
S FireData 

B SoundPropagationMap 


25280 

11676 

111767 

61220 

7697 

583 

1623 

7755 





lame 

Budget 

Used Size 

, (Budget context: default) 

-2.048 00 

0 00 

l ■ □ Unclassified 

150 00 

107 02 

J £ i (Data) 3D - Meshes 

266 00 

273.39 

1 (Data| 3D - Particles 

3200 

4224 

1 EJ (Data) 3D Textures 

1.443 00 

1,370.09 

T extuxes 

1.443 00 

1,370.09 

va> Graphic ::PhysMem::Texture::RC_Unknown 


13019 

L - Graphic PhysMem: Texture: :RC_Urtknowrr: 


0 09 

+ 0 “Graphic PhysMem Texture RC_Envwonment“ 


820 38 

+ 0 ‘Graphic PhysMem Texture RC_Character“ 


259 94 

+ 0 “G raphic PhysM em : T exture R C J llegal" 


26 00 

± 0 "G raphic PhysM em: : T exture R C_FX* 


79 69 

1 0 “Graphic .PhysM em.: Texture RCJnterf ace* 


038 

1 0 “Graphic .PhysMem. Texture GFXTextureArrayMa. 


53.44 

0 *G raphic PhysM em : T exture R C_G ameplay* 


0.00 

] (Data) Animation 

190 00 

294 15 

[ 1 (Data) Destruction 

200 00 

282.35 

| j (Data) Entities 

80 00 

90 87 

i [J (Data) Localization 

20 00 

5 90 

[ j |Data| Sound 

700 00 

81039 

J £_) (Engine | 30 - Buffers 

21200 

183 60 

j (Engine) 3D Misc 

300 00 

465 85 

[ j (Engine) 3D - Textures 

373.00 

375.06 

] Qj [Engine] Animation 

55.00 

53 92 

| [Engine] Debug 

0.00 

116.18 

j □ [Engine] Destruction 

150 00 

148 54 

[ j [Engine] Fire 

25 00 

37 64 

J [J [Engine] Physics 

220 00 

270 86 

LJ [Engine] Pilot 

55 00 

50 58 








GDC 

Telemetry Monitor 

Connections File Loq 

| 
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Show only worst frame 


Show reversed callstack 


ieVanimation\atom\atomgraphnode-cpp:827) 


a scimitar:SimpleStnngTemplate<char,l>: Assign »0x21A 

► scimitarAtomGraphNode::LlpdateEditorProfiler-i-10DB27F (f:\w\l\osborn\main\framework\source\scimitar\engin 

t scimitar JKUtils:AdclllCSofveBonelKJofc-0xD5 

t somitar:Woridlntormation::SetLastF XU printed ♦ 73518B (fAwM\osbom\maan\framework\source\scimitair\engine\worldinformatton.cpp:77) 

I scimitarrCharacterTaskrUpdateAni mPrePtiysics_Equipment + B FAC2C (f:\w\l\osbom\main\frameworlc\source\scimrtar\engine\scheduleAcharacterscheduierfpfK404) 
t scimitarJKUtik::AddlKSohreBonefKJoto»0x91 

t> scimitartSimpleStnngTemplate <char, 1 > ::oper ator ♦ = +4633D6 (f:\vv\l\osborn\rnain\franriework\socirce\sci mitar\system\containers\simplestnng.cpp:129) 

I scimitardKUtih; AddIKSolveBonelKJob<-Ox 113 
a scimitar :PhysMemTracker<3etlnde)tAodContext • 0x119 

a scimitar :Pt'iysMemTracker:JnternalAdd'»2306A5 (f:\wM\osbom\main\framework\source\scimitar\system\meTnory\physmemtraclcerxppcl21) 
a scimi tar :PHysMem Tracker Add • 22FDF0 (f:\w\l\osborn\main\fTamework\source\scimitac\systcm\memory\physmemtracker.cpp:36) 

a scimitarrDXllResoijrceBciffef Base: Create ~ 191 11 15 (f:\wVlVosbom\main\framework\scHjrce\scimitar\graphic\gfx\dxll\dxllresourcetoufferxppc49) 

t somitanDXl lConstantBuffrr:OXl IConstant Buffer • 1912092 (ft\w\l\osbom\main\fraiTiework\source\scimita«\graphic\gfx\dxllVdxllresourcesxpp:193) 
a scimitar:DXllVertexBoffer:DXHVertexBoffer-r 1911B60 (f:\wM\osbom\main\framework\soorce\samitar\graphic\gfx\dxll\dxllresoorces.cpp 68) 
a scimitar:d3Xl 1 Oevicei.CreateVertexBuMer »0xD9 

t scimitar::GFXStaticMesh::C reateRegularVBAndlB + 0x6 B 

a scimitar:X3FXDynamicSubMeshlnstance::GFXOynamKSubMeshInstance-Ox4Al 
a scimitan:GFXGraph»cObjectFdctory::CreateObjectlnstance*Ox58E 

a scirmtar::Giaphit ManagerCteateHardwafeGraphicOtijeclInstarr e»0xl9 


2581659 

766224 

766224 

435929 

186060 

123218 

109995 

20653 

18554 

15450 

15450 

9802 

9774 

9700 

38 






43945110 

9722879 

9722879 

4795219 

4733685 

7269862 

1209945 

954796 

204094 

197485 

197485 

104576 

103662 

102644 

518 

518 

328 

172 

172 

172 


t sc im ita r::E KGr idGeomet ryirGenerateGnd -«-Qxl3 1 
r SCI mi ta r: :DX 1 1 1 ncJexBcrffe r :DX 1 1 1 rsdexBcrffer ♦ 1911E68 (f:\wVl\osborn\main\framework\soorce\scimitar\graphic\gtx\dxl l\dxl lresourcesxpp:158) 

I scimita r:DXllWinTexture: KSraphicMemTrac leer Add «192B5B8 (f:\wM\osbom\main\framework\socirce\scrmitar\graphic\gfx\dxll\dxlltextijre wm.cpp:384) 
k scimitar:PhysMcfnTrackerc:PtacementAdd ♦ 230012 (h\w\l\ostoom\main\framework\source\scirnrtar\system\memory\phys»iriemtmckerxpp:107) 
scimitar JKUti Is: AddlKSohre Bond KJoto »0xCF 

scimitar :PlayerinfoFireDataProviderItem::P1ayerInfoFireData Providerltem » 2932257 (£\w\l\osbom\ma<n\rainbowsix\source\scimitar\ra«nbowsix\fire\dataprovidef\p<ayennfofiredataprovider.cpp:114) 

scimitar :Fiie_HudMPRocindsInfos::DoPfepaiefireElementData-292057A (F\w\l\osborn\main\rainbowsix\source\scimitar\rainbowsix\fire\hiKfVmp\roondsinfos\fire_hodmproundsinfc>s.cpp:323) 

scimitar:E ircPlayersStatusTcamListDataProvidor BuildPlayerltem ♦ 0x2A4 

scimitar -FirePlayersStatusTeamLrstDataProvider; BuildPlayerJtem *0x2 EX) 

somitar:SoundVisualEventManager::ProcessConstruct ♦ 0x1 1A 

scimitar :SourvdVisual Even tManager:ProcessC on struct ♦ 0x140 

somitorrLSimpleStnngT oUTF 8SimpleStnng ♦ 1C769E (E\w\l\ostoom\mairi\fromcwork\source\scimitar\systemVcontainers\s«mplestnng.cpp25) 

scimitarrOnl ireFileWriterlmpUnternal Wnte * 1AB436F (f.\w\l\osbom\main\framework\source\scimitar\onlinemodule\framework\pnvate\File\onlinefilewriterxpp:310) 

scimitar: T ogValoeList:KjetTagNames-r B65327 (f:\wM \ostoorn\moin\framework\source\scimitar\engane\tag\tagxpp:1990) 

sdmitar:SoundResource::ToStnngaOK35 

sdmitar=SoundResource::ToString-r0K5A 

scimitar:SoondInstanceLimiterMarw>9er:ProcessConstnjct«C1457B (F\w\l\osborn\main\framcwork\source\scimitar\engine\5ound\core\soundinstar>celimitecmanager.cpp:34z) 
scimitar:SoundObjectMarsager:CreateSoundObjectPrrvate-rB9DAEE (fr\w\l\osborn\main\framewo»k\source\scimitar\engine\scHjrsd\core\soundobjectmanager.cpp:178) 
scimitar FXlnstarscedn it ♦ F38FE4 (6\«vM\osbom\main\framework\source\scim<tarNengine\fx\fxinstance.cpp:368) 

scimitariAIParameterGetName* 12FB557 (f:\w\l\osborn\mam\frameworkYsocjrce\scinntar\enqi ne\ai\aiparameter\aiparametef.cpp:202) 
scimitar:SimpleStnngTemplate<char,l>:itartsWith» 593611 (t\w\l\osbom\mam\fr a mework\soorce\scimitar\systemVcontair»er5\simplestnng.cpp2S8) 
scimitariAIParameterGetName *Oxl3C 

scimitar:Rig»dBody::CrrateSDKRjgidBodyArxlAddToWorld-r B9723F (f:\w\l \osbom\main\framcworfc\source\scimitar\rngine\physK\ng>dbodyxpp:1864) 

scimitar:R60estructionDataMet :Set ♦ 2387B90 (f:\wVT Vosbom\main\rainbows«x\soorce\sc»mitarVa«nbowsix\gamesystemsVlestnjcbon\r6destructiondatanet.cpp:163) 

sc imitar Dynamic BarkPathData::PathSolv'er::ResolveModifie«Re<ursrvely-26D5.?8F (f:\w\l\ostoorn\mamVainbowsucYso<jrce\scxTiitarVa«nbowsix\a«\barks\barkreqoestrpp:969) 

scimitar:E>yTvamicBarkPathData=PathSolver::ResotveModifirrRecursrvHy * 26D52A3 (f:\w\l\osbom\main\rambowsix\source\scimitar\rainbowsix\ai\barks\barkreqtiest.cpp971) 

scimitar :Scxj ndLocaluedVoiceManagen:ConsUuctDebugVoiceRequest->-0x20C 

somitar:SoundtocalizedVo«ceManagen£onstructDebogVoiceRequestr(bc21F 

so mi tar SocindLocalizedVoice Manager: Prior itySystem:: Exec uteln ter naJ -*-0x706 

scimitar:VoiceOebugger::VotceRequest::Construct«C3ElD5 (E\wM\osbom\main\6ameworkVsource\scimit»rNengine\soond\tool\vo«cedebijggerxpp:20> 

scimi tar iSoundLocalLzedVoiceManager.GonstnjctDefcKX) Voice Request <- 1E75EF8 (f:\w\l\osborn\main\rainbowsix\socirce\scimitarVairibovvsu<\sour>cfYsoijndlocaiizedvc>«cernanager.cpp:490) 
scimitar:SoundlocalizedVo<ceManagen:ConstructDebugVoiccRequcst-*-lE7SF19 (fr\w\1\osbocn\main\rainbc»wstx\saurcr\.%cimitar\rainbownx\sound\soundlocalizedvoicemanager-cpp:491) 
scimitar:SoundLocalizedVoiceManagerConstnjctDebugVoK:eRequest»lE75F3A (t\w\l\osborn\main\rainbowsix\so<jrce\sc>TirtafVainbowsix\sound\soundlocaJizedvoicemanagercpp:493) 
scimitar:SoundlocalizedVo4ceManager:Constn«ctDebugVoiceReguest-rlE75F66 (E\w\lVosbom\maan\rainbowsix\sourceVscimitarSrambowsix\sound\soundlocalizedvoicemanagerxpp:494) 
somita rz Duplication Manager :DuplicateName • 0x7B 

scimitar:NavigateMoveToDestination.~Dup4icate-i-2C264DD (f:\w\l\osbornVnain\rainbowsix\source\scimitai\rainbowsix\a*\enbty1ogic\navigate\navigatemovetodata.cpp295) 
somita rDest ruction Compound! ragmencAddToWorld • 0x1 EA 

.+AAC083 Ef:\wVl\osbom\main\framewortc\source\scimitar\enatne\coreMocalizabontaaconficixoD:492) 
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Counters | 


Connections Service Status Error List 


X 0 A 0 i 0 


Connections count 0 





r Jndex 

tags 

# threaded 

r _kl 
r _type 

f message_string 
r session_id 
O time 
r type 


i 


r6telemetrylogs 



Selected Fields 
9 line 


1000 000 

r file 

i 

800.000 

r message 

600 000 

r category 

o 

400.000 
200 000 

r username 


0 

Available Fields 
Popular 

□ 



August 12th 2015. 15 43 45 831 - October 1 1th 2015. 15 43 45 831 — bv dav 


2015-09-12 

time per day 


Time ^ 


message 


October 9th 2015, 15:14:11.448 [DisabledComponentUpdated] [QueueComponent] Requested update for 
Component Acti veGroundDrone which was inactive. Owner -{object : 
UNKNOWN id-OxfOOOa027 class-€ntity} 


» October 9th 2015, 12:40:19.171 


Allocation 

Type [0] peak allocation [9] waste [247] 

Type [1] peak allocation [0] waste [8] 

Type [2] peak allocation [0] waste [0] 

Type [3] peak allocation [116] waste [4384] 

- at illocati Btc fWil 


e:\rambow6\osborn\raain 
\framework source sci 
mi tar \engi ne\ani mat i o 
n\atom\atominstancepoo1 . cpp 


October 9th 2015, 12:40:19.168 This class has been tagged with the skipreleaseresources mold 
attribute. ReleaseResourcesO should not be called. 

1. Check if the class should really have this attribute(see 
progrmnmer guide for guidelines). 

2. If still needed, add the attribute to the referenced property. 


e : \r ai nbow6\os bor n \mai n 
\framework\source\sci 
mitar\engine\debug\re 
pi ay\debugr epl aydebug 
displaytrack. cpp 


► October 9th 2015, 12:40:18.055 Shut down () 


e : \r ai nbow6\os bor n \mai n 
\framework\source\sci 
mi tar \gr aphi c\ei kon\e 
kshaderdatabasemanager . cpp 


» October 9th 2015, 12:40:18.051 Pilot shutdown 


e : \r ai nbow6\os bor n \mai n 
\framework\source\sci 
mi t ar \engi ne\ai \pi 1 ot 


12,914,781 hits 


.li._iilli_ nil. ■■■_. ■■!■■■ ■ 1.1 lill. ■! 


username 

mdepascale 


sorin. ivascu 


731 sorin.ivascu 


412 sorin.ivascu 


476 sorin.ivascu 
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Telemetry Uses 

• Performance Metrics 

• Spikes Detection 

• Load Time Metrics 

• Startup Time Metrics 

• Compile Time Metrics 

• User Focus Metrics 

• Centralized Logs 

• Memory Tracking 


Cases 

• Buffers/Pools Size 
Tracking 

• Used Assets/Localization 
Tracking 

• Network Replication 
Debugging 

• Bandwidth/Latency 
Metrics 


Editor MTBC Stats 

lint; 
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1 

50.92 CHALET TDM - BOMB PS4 












! 

51.93 KAFE DOSTOYEVSKY TDM - SECURE AREA PS4 












52.13 KANAL TDM - HOSTAGE PS4 













52.39 HOUSE TDM - BOMB PS4 












1 

55.37 YACHT TDM - HOSTAGE PS4 












56.52 HEREFORD BASE TDM - BOMB PS4 












1 

57.40 HEREFORD BASE TDM - SECURE AREA PS4 












59.86 CONSULATE TDM - HOSTAGE XBOXONE 












59.98 CONSULATE RESCUE HOSTAGE PS4 












60.47 HOUSE HOSTAGE XBOXONE 













60.53 HEREFORD BASE SECURE AREA XBOXONE 













60.66 YACHT TDM - BOMB XBOXONE 












1 

1 

60.83 HEREFORD BASE CONTAINMENT PS4 












61.23 YACHT TDM - HOSTAGE XBOXONE 












61.66 OREGON TDM - HOSTAGE PS4 












61.76 PRESIDENTIAL PLANE SECURE AREA XBOXONE 













61.88 CHALET RESCUE HOSTAGE PS4 

1 












62.32 KANAL TDM - BOMB PS4 













62.74 PRESIDENTIAL PLANE TDM - SECURE AREA PS4 












i 

62.80 OREGON SECURE AREA XBOXONE 













IPercentage of Frames below 60 FPS Leaderboard (last day) 

97.98 HOUSE TDM - BOMB PS4 















■ 

95.65 CLUBHOUSE TDM - HOSTAGE PS4 















■ 

90.90 YACHT TDM - SECURE AREA PS4 
















l 

89.24 KAFE DOSTOYEVSKY TDM - SECURE AREA PS4 
















1 

■ 

83.80 HEREFORD BASE TDM - BOMB PS4 


















74.97 HEREFORD BASE TDM - SECURE AREA PS4 
















' 


73.99 KANAL TDM - HOSTAGE PS4 















1 

62.70 CHALET TDM - BOMB PS4 















1 

54.79 YACHT TDM - HOSTAGE PS4 















1 

51.14 OREGON TDM - HOSTAGE PS4 















1 


48.40 YACHT TDM - BOMB XBOXONE 


















46.56 KANAL TDM - BOMB PS4 


















44 70 CONSULATE TDM - HOSTAGE XBOXONE 


















42.06 YACHT TDM - HOSTAGE XBOXONE 


















40.80 PRESIDENTIAL PLANE TDM - SECURE AREA PS4 


















34.73 KAFE DOSTOYEVSKY TDM - BOMB PS4 


















33.53 HEREFORD BASE TDM - HOSTAGE XBOXONE 
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Kibana 


Discover Visualize Dashboard Settings 



0 Last 30 days 

Q. a B & t? O O 


M Anvil Last Week MTBC Minutes ✓ X 

W Anvil Last Week Crash Count ✓ X 

538.246 

Average meantimebetweencrashesm 

120 

Sum of abnormal termination count 

^ 1 



l»ld Anvil Crashes per Day Segmented 

II I' 


Anvil Average Sessions per Day 


1^ Anvil MTBC per Day 


77.517 


Average instancecount 


§ 500 
® 400-1 

I 

.8 300 

l| 200 

c 100 


✓ X 
Legend O 

• Sum of anvil_crash_c... 

• Sum of scimitar_crash. . 

• Sum ofguildlib_crash. 



stat begin tme per day 



| 1 ^ J | 

A 

</> Anvil Global Dashboard Legend ✓ x 

Anvil Crash Count per Day ✓ X 

The Anvil MTBC is computed with the running time capped to 8 hours and 
includes the Scimitar Toolmode and Guildlib crashes. 

I 60 - 

3 

1 40 - 

■3 

J 20- 



1 0- 



2016-01-22 2016-01-26 2016-01-30 2016-02-02 2016-02-06 2016-02-10 2016-02-14 

stat begm tirne per day 


A 
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Agenda 


• Intro 

• Unified Telemetry Rationale / Desig 

• Rainbow Six Siege Telemetry 

• Postmortem 
. Q&A 


Definition of Telemetry 

" Telemetry is the highly automated 
communications process by which 
measurements are made and other data 
collected at remote or inaccessible 
points and transmitted to receiving 
equipment for monitoring." 


source: Wikipedia 
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Definition of Telemetry 

Telemetry is 

1 1 CtcJ I I I veil tQ I 1 1 Vwt 1 1 1 1 iO) ,0) [O) 

measurements i 

collected it remote 


automated 
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Use Cases 

• Stats Gathering 

• Events 

• State Snapshots 

• Live Debugging 



GDC 




GAME DEVELOPERS CONFERENCE March 14-18, 2016 • Expo: March IE 


Current Tech 

• Logging Systems 

• Tracing Libraries 

• Metrics Collection 

• Memory Profilers 

• Performance Profilers 

• Physics Debugger 

• Animation Debugger 


(file) 

(file) 

(network) 

(file) 

(file) 

(live tool) 
(live tool) 
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The Need for Unified Telemetry 

Scattered Tools 
Scattered Data 


Scattered Knowledge 



Non Uniform Telemetry Data 



metrics 


animation data 


ENGINE 


V 


gameplay events 


gpu data 



perf_capture.fap 


mory_dump. memento 


Anim 

Debugger 

Profiler 
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Multiple Processes 


4 


metrics 


Usage metrics 


gameplay events 



nimation data 


nA 


RPC activity 



mouse events 
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Large Teams / Multisite Develop 

V r zm& ® 






a* 








IB, 
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Unified Telemetry 
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Unified Telemetry 

i. A single pipe for all telemetry data 
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Unified Telemetry Stream 


ENGINE 


metrics ogging 


perf nemory ameplay iim ipu 

amples state events ata ata 
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Unified Telemetry 

1. A single pipe for all outgoing telemetry 
data 

2. Universal timestamping / ID 


Stream 2 


Stream 1 
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Unified Telemetry 

1. A single pipe for all outgoing telemetry 
data 

2 . Universal timestamping / ID 

3. A scalable infrastructure to 

1. Collect 

2 . Process (Custom) 

3. Store 

4. Retrieve 


ink 
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Telemetry Infrastructur 




Perf 

Profiler 


Memory 

Profiler 


Live 

Debugger 








Current Tech, Extended 


• Logging Systems 

• Tracing Libraries 

• Metrics Collection 

• Memory Profilers 

• Performance Profilers 

• Physics Debugger 

• Animation Debugger 


(telemetry, file) 
(telemetry, file) 
(telemetry, network) 
(telemetry, file) 
(telemetry, file) 
(telemetry, live tool) 
(telemetry, live tool) 
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Benefits 

• Simpler Tools 

• Cross Domain Analysis 

• Team Wide Analysis of non-stats Data 


Easier Collaboration 


GDC 
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Agenda 


• Intro 

• Unified Telemetry Rationale / Desig 

• Rainbow Six Siege Telemetry 

• Postmortem 
. Q&A 
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Telemetry Stream 

Lossless 
Ordered 
Stream of 
Heterogeneous 
Binary Encoded 
Events 


C++ Client 


• Layered 

• Engine/platform agnostic base (C+ + 11) 

• Engine/Platform integration code (PC, PS4, XB1) 

• Completely Passive 

• No memory allocations 
. No implicit I/O 

• Modular & Configurable 

• Runtime Bit Mask per class of events 
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Telemetry Primitives 


• Counter 

• Value 

• Constant 

• GraphSettings 

• Marker 

• Scope 

• Log 


BinaryData 

TextData 

SendFile 

Task 

Resource 

Processlnfo 


Telemetry Primitives (Scimitar) 


• CPU Frame 

• GPU Frame 

• Engine Update 

• Graphics Update 

• Profilerstreaming 


MemoryEvent 

Userlnfo 

CodeDataSound 

Tainted 
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Telemetry Primitives (Rainbow6) 

• Match 

• Round 

• Map 

• GameMode 


GameType 

GameplayEvent 

LocalizationEvent 
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C++ API Usage Examples 

//track destructible entities creation 

TELEMETRY_HI_FREQ_COUNTER ( " [Engine] [EntityCreate ] scimitar: : DestructibleComponent " ) ; 
//Generate Graph data for latency 

TELEMETRY_GRAPH_SETTINGS ("Name=latency; Unit=ms; Color=red; Group=Network; " ) ; 
TELEMETRY_VALUE ("latency", sampledLatency ) ; 

//Record connected user 

TELEMETRY_MARKER_FORMAT ( " [Gameplay ] [Network] user connected {0}", username); 

//Track loading time for maps 

{ 

TELEMETRY_LOW_FREQ_SCOPE_FORMAT (" [Loading] Map {0}", mapName); 

FastLoadData (mapName) ; 


} 
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C++ API Usage Examples 


//Save profiler snapshot and collect file centrally 
auto&& const filename = Prof Her : : GetFilename ( ) ; 

Profiler: : SaveToDisk (filename) ; 

TELEMETRY_SEND_FILE (filename) ; 

//Record settings after recursive inclusion of all ini files 
TELEMETRY_TEXT_DATA ( " IniSett ings " , settings . ToString ( ) ); 

//Send Begin/End Frame Markers 

SCIMITAR_TELEMETRY_BEGIN_CPU_FRAME (frameNuraber) ; 

SCIMITAR TELEMETRY END CPU FRAME (); 


//Send Gameplay specific info 
R6_TELEMETRY_BEGIN_R0UND (mapNarae) ; 
R6 TELEMETRY END ROUND ( ) ; 



■ Ilk 



C++ Client Initialization 


• Init 

• Very first line in main() 

• Uses provided temporary buffers 

• Connect 

• Called after engine has been initialized 

• Establishes connection (or setup file output) 

• Flushes data and switches to normal behavior 
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C++ Client Performance 

• Queues of buffers 

• 16MBs Total for RainbowSix 

• Lockless access to buffers 

• Uses a Read-Copy-Update strategy 

• Hierarchy of TLS buffers 

• When global ordering is irrelevant 

• Ammortized Cost per Global Primitive: 

. RDTSC + 3 CMPXCHG + REP MOV 


lllk 
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C# Client 

• A Mirror of the C++ API 

• Less performance obsessed 

• Pure C# 

• no P/Invoke 

• no managed C++ 

• Supports Async Data Transfer 

• Uses [Conditional] to be compiled out 

Ilk 
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C# API Usage Examples 

//Collect file 

Telemetry . SendFile ("user.ini") ; 

//track focus switch 

Telemetry . Counter (" [Anvil ] [Tool] Focus Switch", +1); 

//Record events 

Telemetry . Marker (" [Anvil ] [Tool] Plugin {0} Loaded", pluginName) ; 
//Track loading time for maps 

using ( Telemetry . TimingScope (" [Anvil ] [Loading] Map {0}", mapName) ) 

{ 

Scimitar . LoadWorld (mapName) ; 

} 
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Network Usage 


Live 

Debugger 



i— 18, 2016 #GDC1B 



I 
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Adaptive Scalability 
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Telemetry Receiver 

• Receives data fast 

• to avoid buffer overruns on client 

• Dispatches deserialized data to Processors 

• Used as 

• Local Receiver 

• Global Receiver 
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Telemetry Processors 

• Aggregate Data 

• Analyze Data 

• Transform/ Forward Data to visualizers 

• (e.g. Local Tools) 

• Store Data into DB 
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C# Processor Examples 

public class CountersStreamProcessor : TelemetryStreamProcessor 

{ 

void OnEvent (StreamOpenEvent evnt, Guid guid) ... 

void OnEvent(EndOfPrologueEvent evnt, Guid guid) ... 

void OnEvent(StreamClosedEvent evnt, Guid guid) ... 

void OnEvent (ProcessInfoEvent evnt, Guid guid) ... 

void OnEvent (AnvillnfoEvent evnt, Guid guid) ... 

void OnEvent (ScimitarlnfoEvent evnt, Guid guid) ... 

void OnEvent(EndOfProcessingEvent evnt, Guid guid) ... 

void OnEvent(RecondNameMappingEvent evnt, Guid guid) ... 

void OnEvent(CounterEvent evntj Guid guid) ... 

void OnEvent(RecondSlotMappingEvent evnt, Guid guid) ... 

void OnEvent(HighFrequencyCountersEvent evnt., Guid guid) ... 

void OnEvent (BinaryDataEvent evnt, Guid guid) ... 

void OnEvent(TaintedDataEvent evnt, Guid guid) ... 


} 
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Scalability 

• Optional Local Telemetry receiver 


• Global Telemetry receivers 

• Centralized NoSQL Database 
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Infrastructure Performance 

• Local Telemetry Receiver 

• Real-time processing on typical workstation under 
typical load (i.e. 1 Editor, 1 Engine) 

• Global Telemetry Receiver 

• But more streams (500~1000 on Rainbow Six) 

• Up to 30~60 minutes behind, during busy periods 

• Peak on Rainbow Six: 300+ GBs / day 


lllk 


GDC 




GAME DEVELOPERS CONFERENCE March 14-18, 2016 • Expo: March 16-18, 2016 #GOC16 


Hardware Spec 

• Intel XEON 

• E5-1650 v2 @ 3.50 Ghz 

• 32/64 GB RAM 
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|| Telemetry Server Sta * 




Ct 0 r6-mongodb.ubisoftorg:5601/#/dashboard/Telemetry-Server-Status?_g=(refreshlnterval:(display:'1 5%20miruites',pause:!f,section:2/value:900000),time:(from:‘2015-10-06T00:52:59.439Z/node:absolute,to:‘2015-10-07T04:21 :35.327Z))&_a=(filters:!0,panels:!((col 1 = 


Telemetry Server Max Connections 

4,000 


✓ x Telemetry Server Max Connections Counter 



Legend O 

• Max connections 

• Max sources 


Max connections 


time per 30 minutes 


Telemetry - Connections over time (by source process) 


✓ x Telemetry - total events 

Legend O 


ii ■ilnllLi 


02 00 OS 00 



• Chisel 

• Anvil 

• Tool 




1 , 608 , 960,111 


Sum of total events 


Telemetry - Connections by Stucfto over time 


✓ X T eiemetry - total bytes 


II 


lllllllllillllli 


23 00 02 00 06 00 08 00 110C 

begm tlme per 30 



101 , 821 , 922,835 


Sum of total_bytes 


I ; E- i. 


O 


F:\R6 Te... Teleme.. 


HP R6TELE... TaskM... F:> 


F:\R6 Te... 


□ Teleme... 


a - • © 


23:43 

2015 - 10-25 
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Data Visualization 

• Custom Tools 

• Kibana 

• Tableau 

• Custom Web Interface 
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GearStudio 



® GEAR STUDIO 20 i 5 .ia 09 -ii 2346 .jiow.fri.- File Edit View Window Help Plugins 



lllk 
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Kibana 

E arns 


J ■ Compile Stats Dashi * ^ 


II - l S ■ 


C Q r6-mongodb.ubisoftorgr6i;n .jt vr_ ,1 i.oi ; l,i, '1 r.%2 jir.jriijt,?: 'f ;,r t jl.j.;- '>00 X fn.t nx-i-i .min nx-X ■:[ r.uAXXM, )X_.j = fi x-i : 'Q.pjr j S x I : :t.r: ~ :j: 5 p: ixtxvi-T'tj’ 5 


Discover Visualize Dashboard Se tings 


message 
category 
username 
Available Fields 


message.string 

sessionjd 


August 12th 2015, 15:43 45 831 - October 1 1th 2015 


Ll...nih. 


Lll. 


!■>.. illli. 


October 9th 2015, 15:14:11.448 [DtsabledComponentUpdated] [QueueComponent] Requested update for 
Component ActiveGroundDrone which was inactive. (X*ner-{object: 
UNKNOWN id-0xf000a027 class-Cntity} 


October 9th 2015, 12:40:19.171 ••••••••••••••••••••••••••••••••••••••••••••••••••Kino p, 

Allocation 

Type [0] peak allocation [9] waste [247] 

Type [1] peak allocation [0] waste [8] 

Type [2] peak allocation [0] waste [0] 

Type [3] peak allocation [116] waste [4384] 


2:40:19.168 This class has been tagged with the skipreleaseresources mold 
attribute. ReleaseResourcesO should not be called. 

1. Check if the class should really have this attribute(see 
programmer guide for guidelines). 

2. If still needed, add the attribute to the referenced property. 


October 9th 2015, 12:40:18.055 ShutdownQ 


2:40:18.051 Pilot 


Top compilers end top compied projects 


• projectsvealueisWIL 
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Web Interface 





GDC 




GAME DEVELOPERS CONFERENCE March 14-18, 2016 • Expo: March 16-18, 2016 #GOC16 



Centralized Data Storage 


• MongoDB 



mongoDB 


• main storage, documents and files (GridFS) 


• ElasticSearch 



replicated data, consumed by Kibana 


elasticsearch 
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Offline Support 


• Just save to disk in a receiver is not 
available 

• e.g. XLC 

• Each Receiver can load and process 
offline streams 


Wire Protocol 


• Binary Encoded 

• Memcpy, no endian-swap on x64 

• Frames (MessagelD, Length, Payload) 

• Can skip unsupported events 

• Timestamps not part of the protocol 

• Even though is in almost every message type 

• Supports 7bits encoding 

• But we never had to use it 


GDC 
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Agenda 


• Intro 

• Unified Telemetry Rationale / Desig 

• Rainbow Six Siege Telemetry 

• Postmortem 
. Q&A 


GDC 
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Telemetry Uses 

• Performance Metrics 

• Spikes Detection 

• Load Time Metrics 

• Startup Time Metrics 

• Compile Time Metrics 

• User Focus Metrics 

• Centralized Logs 

• Memory Tracking 


Cases 

• Buffers/Pools Size 
Tracking 

• Used Assets/Localization 
Tracking 

• Network Replication 
Debugging 

• Bandwidth/Latency 
Metrics 


Editor MTBC Stats 

lint; 
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Postmortem (Pro) 

• Simpler Tools 

• Data Cross Correlation 

• More Effective Collaboration 


Go get a Unified Telemetry System! 
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Postmortem (Fails) 

• Sockets 

• Lesson Learned: don't use blocking sockets from non-background threads 

• Optional Timestamp 

• Lesson Learned: bandwitdh never an issue, add timestamp to protocol 

• C# Server 

• Lesson Learned: overspec for memory 

• Servers Hardware Setup 

• Lesson Learned: don't run out of space on Mongo 


I I Ik 
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Postmortem (Reccomendations) 

• Know your Questions before Tracking 

• Tracking is costly. Only do it if necessary 

• No Tracking without Owner (not you) 

• Both tracking and data rots quickly 

• Having Data is good 

• Having too much data is bad 

• Having bad data is a dangerous 


lllk 


What's Next 


• Stress tested in full production cycle 

• Rainbow Six Siege 

• Unannounced Project 

• Close the loop 

• Record/Replay/Automated Testing 

• Internal Network Only 

• yet multisite (North America, Europe, Asia) 

• Would it work on the Internet? 
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Special Thanks 

• Ubisoft 

• The R6 Team 

• The R6 Tools Team 

• Mark Besner 

• Gabriel Langelier 


• Florent Jousset 

• Ouamer Dahmani 

• Jean-Francois Richard 

• Sebastien Lussier 

• Julien Merceron 



Thank You! 
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GOC ? 

Questions? 

maurizio.depascale@ 
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